Lessons from the Congested Clique Applied to MapReduce

The main results of this paper are (I) a simulation algorithm which, under quite general constraints, transforms algorithms running on the Congested Clique into algorithms running in the MapReduce model, and (II) a distributed O(∆)-coloring algorithm running on the Congested Clique which has an expected running time of O(1) rounds, if ∆ ≥ Θ(log n); and O(log log log n) rounds otherwise. Applying the simulation theorem to the Congested Clique O(∆)-coloring algorithm yields an O(1)-round O(∆)-coloring algorithm in the MapReduce model. Our simulation algorithm illustrates a natural correspondence between per-node bandwidth in the Congested Clique model and memory per machine in the MapReduce model. In the Congested Clique (and more generally, any network in the CONGEST model), the major impediment to constructing fast algorithms is the O(log n) restriction on message sizes. Similarly, in the MapReduce model, the combined restrictions on memory per machine and total system memory have a dominant effect on algorithm design. In showing a fairly general simulation algorithm, we highlight the similarities and differences between these models.